[BugFix]add all2all when dp_size > 1 && downgrade npu_dequant_swiglu_quant #819
+313
−76
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
May 12, 2025 21:42
Signed-off-by: angazenn <zengyanjia@huawei.com>
Signed-off-by: angazenn <zengyanjia@huawei.com>
Signed-off-by: angazenn <zengyanjia@huawei.com>
Angazenn
force-pushed
the
all2all
branch
3 times, most recently
from
af2215a to
d6cf7a1
Compare
Angazenn
changed the title
Angazenn
force-pushed
the
all2all
branch
2 times, most recently
from
b2ca756 to
e0ab8e0
Compare
added 4 commits
May 13, 2025 14:18
Signed-off-by: angazenn <zengyanjia@huawei.com>
Signed-off-by: angazenn <zengyanjia@huawei.com>
Signed-off-by: angazenn <zengyanjia@huawei.com>
Signed-off-by: angazenn <zengyanjia@huawei.com>
| dist.all_to_all_single(gather_sizes, | ||
| scatter_sizes, | ||
| group=ep_group.device_group) | ||
| scatter_size_list = scatter_sizes.cpu().tolist() |
There was a problem hiding this comment.
This may introduce serious performance regression, please note that we will change this in future.
| gather_dim, scatter_sizes, | ||
| gather_sizes) | ||
|
|
||
| def reduce_scatter(self, |
There was a problem hiding this comment.
Remove this if you do not use it.
vllm_ascend/models/deepseek_v2.py
Outdated
| attn_metadata = get_forward_context().attn_metadata | ||
| if attn_metadata is None: | ||
| # when profile runs, force experts load balance to avoid high memory | ||
| # consumption from 1 rank. |
There was a problem hiding this comment.
Add more comments on this
added 2 commits
May 13, 2025 22:37
Signed-off-by: angazenn <zengyanjia@huawei.com>
Signed-off-by: angazenn <zengyanjia@huawei.com>
Angazenn
force-pushed
the
all2all
branch
2 times, most recently
from
3349fbd to
30eafb9
Compare
Signed-off-by: angazenn <zengyanjia@huawei.com>
Angazenn
changed the title
ganyi1996ppo
approved these changes
May 15, 2025
yiz-liu
added a commit
to yiz-liu/vllm-ascend
that referenced
this pull request
May 17, 2025
…_swiglu_quant (vllm-project#819)" This reverts commit 1e67089.
|
This is the best solution for A3 performance. Is there a best solution for A2 performance? |
ganyi1996ppo
pushed a commit
that referenced
this pull request
May 24, 2025
### What this PR does / why we need it? This PR fixes two accuracy bugs incurred by PR #819 when running deepseekv3 series models: 1. #819 adds `all_to_all` communication in quantized cases, but `all_gather` && `reduce_scatter` are removed in both of quantized and unquantized cases. When running unquantized deepseekv3 models with `ep_size == world_size`, the moe modules fail to communicate. Therefore, this PR adds `all_to_all` communication on unquantized situation to solve this accuracy issue. 2. Use `ep_size` rather than `dp_size` to decide whether to use `all_to_all` in moe. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? CI passed with new added/existing test. --------- Signed-off-by: angazenn <zengyanjia@huawei.com> Co-authored-by: angazenn <zengyanjia@huawei.com>
momo609
pushed a commit
to momo609/vllm-ascend
that referenced
this pull request
May 30, 2025
…oject#897) ### What this PR does / why we need it? This PR fixes two accuracy bugs incurred by PR vllm-project#819 when running deepseekv3 series models: 1. vllm-project#819 adds `all_to_all` communication in quantized cases, but `all_gather` && `reduce_scatter` are removed in both of quantized and unquantized cases. When running unquantized deepseekv3 models with `ep_size == world_size`, the moe modules fail to communicate. Therefore, this PR adds `all_to_all` communication on unquantized situation to solve this accuracy issue. 2. Use `ep_size` rather than `dp_size` to decide whether to use `all_to_all` in moe. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? CI passed with new added/existing test. --------- Signed-off-by: angazenn <zengyanjia@huawei.com> Co-authored-by: angazenn <zengyanjia@huawei.com> Signed-off-by: wangxiaoxin (A) <w00664509@china.huawei.com>
zxdukki
pushed a commit
to zxdukki/vllm-ascend
that referenced
this pull request
Jun 3, 2025
…oject#897) ### What this PR does / why we need it? This PR fixes two accuracy bugs incurred by PR vllm-project#819 when running deepseekv3 series models: 1. vllm-project#819 adds `all_to_all` communication in quantized cases, but `all_gather` && `reduce_scatter` are removed in both of quantized and unquantized cases. When running unquantized deepseekv3 models with `ep_size == world_size`, the moe modules fail to communicate. Therefore, this PR adds `all_to_all` communication on unquantized situation to solve this accuracy issue. 2. Use `ep_size` rather than `dp_size` to decide whether to use `all_to_all` in moe. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? CI passed with new added/existing test. --------- Signed-off-by: angazenn <zengyanjia@huawei.com> Co-authored-by: angazenn <zengyanjia@huawei.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it?
all_to_allcommunication operator to fixallgatherbugs when dp_size > 1. Besides, it adds a naive implementation of force-load-balance when doing profile runs.npu_dequant_swiglu_quantonly supports input hidden_states with dtypetorch.int32. This tensor occupies space ofglobal_bs * seq_len * topk * hidden_size, which might be very large asep_sizegrows. Therefore we need to disable this operator and use originalswiglu&&quantize.Does this PR introduce any user-facing change?
No.
How was this patch tested?
By performing offline inference:
